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RELATIONSHIPS BETWEEN TEACHER KNOWLEDGE, ASSESSMENT 
PRACTICE, AND LEARNING - CHICKEN, EGG, OR OMELET? 

Joan Herman, Ellen Osmundson, and Yunyun Dai 
CRESST/UCLA 

Cathy Ringstaff, Mike Timms 
WestEd 

Abstract 

Drawing from a large efficacy study in upper elementary science, this report had three 
purposes: First to examine the quality of teachers’ content-pedagogical knowledge in 
upper elementary science; second, to analyze the relationship between teacher knowledge 
and their assessment practice; and third, to study the relationship between teacher 
knowledge, assessment practice, and student learning. Based on data from 39 teachers, 
we found that students whose teachers frequently analyzed and provided feedback on 
student work had higher achievement than students whose teachers spent less time on 
such activities. Our findings support other research indicating the power of well- 
implemented formative assessment to improve learning. 

Introduction 

Spurred by Black and Wiliam’s (1998) meta-analysis documenting formative 
assessment as a powerful classroom intervention, particularly for low achieving students, and 
supported by researchers and practitioner communities from diverse theoretical perspectives 
(see reviews by Shepard, 2005; Herman, 2010; James et ah, 2007), policymakers across the 
world are considering formative assessment as a primary approach to educational reform 
(OECD, 2005; CCSSO, 2008). In the US, billions of dollars have been invested in Race to 
the Top initiatives that put Common Core State Standards, assessment, and use of data front 
and center, including over $350 million awarded to two state consortia to develop new 
standards-based assessment systems. While system development focuses primarily on testing 
for accountability purposes, the federal assessment grants, for the first time, recognize the 
importance of formative assessment and of building teachers’ capacity to use it. 

These are promising developments for pushing formative assessment to fruition in 
classroom practice. Yet at the same time, recent studies reveal challenges in implementing 
quality formative practice (Heritage, Kim, Vendlinski & Herman, 2009; Heritage, Jones & 
White, 2010; Herman, Osmundson, & Silver, 2010); show non-robust results with regard to 
effects on student learning (Eurtak, et ah, 2008; Herman et ah, 2006; Wiley & Ciafolo, 
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2010); and raise questions about the researeh base underlying formative assessment (Bennett, 
2009). Just as the concept of formative assessment underscores the central role of evidence in 
effective teaching and learning, so too do policymakers and practitioners need evidence on 
which to build effective formative practices. 

Fundamentally, formative assessment involves the use of assessment to ^rm” 
subsequent instruction (Black & Wiliam, 2004) - or as the Council of Chief State School 
Officers (CCSSO) defines it, -a- process used by teachers and students during instruction that 
provides feedback to adjust ongoing teaching and learning to improve students' achievement 
of intended instructional outcomes" (FAST/SCASS, 2008). Formative assessment involves 
knowing what the learning goals are, eliciting evidence of student status relative to the goals, 
and taking action to close any gap between students’ current status and the desired goal(s) 
(Black & Wiliam, 1998, 2004, 2009; Black, Harrison, Marshall & Wiliam, 2003; Hattie & 
Timperley, 2007; Heritage, 2010; Sadler, 1989; Shepard, 2005). 

Because formative assessment is a dynamic process of evidence elicitation, analysis, 
and action, it clearly makes demands on teachers’ content and pedagogical knowledge. 
Without such foundational knowledge, teachers’ formative assessment may yield faulty 
decisions that could divert rather than promote student progress. At the same time, there also 
could be a reciprocal relationship between teachers’ use of assessment and their content and 
pedagogical knowledge. Teachers who engage in formative assessment are continually 
attuned to and responding to student learning progress. Educators who analyze student 
learning, consider potential obstacles or misconceptions limiting this learning, and reflect on 
the effectiveness of prior and subsequent next steps — may well deepen their content and 
pedagogical knowledge, particularly if such activities occur in the context of professional 
learning communities (Little, 2003; Stoll et al. 2006) 

While the challenge of teachers’ content-pedagogical knowledge has been documented 
(Heritage et ah, 2009; Heritage, Jones & White, 2010; Herman et ah, 2010), few studies have 
examined the relationship between such knowledge and teachers’ assessment practices, nor 
examined how teachers’ knowledge may moderate the relationship between assessment 
practices and student learning. The study reported here draws from a larger intervention 
study in upper elementary science to explore these relationships. Study research questions 
include: 

1. What is the quality of teachers’ content-pedagogical knowledge? 

2. What is the relationship between teacher knowledge and assessment practice? 



2 




3. What is the relationship between teaeher knowledge, assessment praetiee, and student 
learning? 

Methodology 

While the larger study is based on two eohorts of teaehers aeross three states who 
partieipated in a randomized field study of the effeets of ineorporating new, currieulum- 
based assessments into an upper elementary hands-on seienee eurrieulum program, this study 
is based only on the Cohort 1 sample for whom full data are available. Because the validity 
of the curriculum-based assessments used in the study has been established (Draney et ah, 
2005), study data provide a good opportunity to examine the role of teacher content- 
pedagogical knowledge and teacher assessment use in student learning, without any 
confounding from the quality of the assessment data. 

For each cohort, schools in each state (and the teachers within them) were randomly 
assigned to either treatment (revised program with curriculum-embedded assessments) or 
control (traditional program) conditions. Treatment teachers participated in two days of 
summer professional development to orient them to the new curriculum and assessment, 
follow-up sessions to support the analysis of student work, and a practice year for 
implementing the curriculum in preparation for the Year Two investigation of treatment 
impact. Control teachers also participated in a similar amount of summer professional 
development focused on teaching the original curriculum. All teachers in the study 
implemented two curriculum units, one on Magnetism and Electricity and the second on 
either Structures of Life or Water. 

Given the focus of the treatment, the study used a variety of methods to collect data on 
teachers’ assessment practices, including teacher surveys, logs, and direct measures of 
teachers content-pedagogical knowledge. In addition, for the impact study year, measures of 
student learning were used. Cohort 1 study data are now complete. Cohort 2 teachers are 
completing their impact study year; thus, full data are not yet available. 

Sample 

Cohort 1 is comprised of 39 teachers from a southwest state. Table 1 — ^which shows the 
demographic characteristics of the educators in the study — reveals no major differences 
between treatment and control teachers. 
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Table 1 

Teacher Demographic Information: Cohort 1 



Descriptor 


Control 

A=19 


Treatment 

A=20 


Sex 


Male 


1 


0 


Female 


18 


20 


Ethnicity 


White 


17 


17 


Hi spanic/Latino/a 


2 


2 


Native American/Affican American 


0 


1 


Other 


0 


0 


Highest degree received 


Bachelor’s + credential 


5 


6 


Bachelor’s + credential + units beyond 


3 


4 


Master’s: 


3 


5 


Master’s + units beyond 


8 


5 


T eaching credential 


General elementary 


18 


17 


General secondary 


1 


1 


Special emergency 


2 


3 


Multiple subject 


1 


1 


Single subject 


2 


2 


Bilingual 


4 


6 


Administrative 


1 


1 


Other: (Early childhood, TESOL, guidance, special ed., science endorsement) 


4 


5 


Grade level taught 


grade 


0 


0 


4* grade 


19 


20 
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Descriptor 


Control 

A=19 


Treatment 

A=20 


Years of experience teaching elementary grades 


Average number of years 


12.0 


8.4 


Range of years teaching 


1-32 


2-25 


Years teaching science curriculum unit 


Average number of years 


3.0 


2.6 


Range of years teaching 


1-11 


2-12 


Number of science PD hours in the past 2 years 


Average number of hours 


19.6 


21.3 


Range of hours 


4-100 


2-80 



“Teachers may hold multiple credentials; therefore, the total number of credentials represented in the table 
exceeds the total number of teachers who participated in the study. 



Table 1 highlights that the majority of the study’s participants were white females who 
possessed an elementary credential. It is also evident that control teachers were more likely 
to hold a masters degree than treatment teachers. However, in regards to their average 
number of years of teaching experience, as well as their experience teaching the science 
curriculum under study — control and treatment teachers appeared to be more similar. 

The project continuation or completion rate for Cohort 1 was high; in fact, of the 39 
teachers who began the project in August 2008, 32 teachers (or 82%) remained in the project 
through its conclusion in June 2010. Most teachers who left the project did so because of 
changes in teaching assignments to different grades or non-project schools (personal 
communication, M. Tiu, July, 2010). 

Study Variables and Instrumentation 

Study variables include teachers’ content-pedagogical knowledge, as measured by 
direct assessment of teachers; teachers’ use of assessment, as measured by teacher logs; and 
multiple, direct measures of student learning. 

Teachers’ content-pedagogical knowledge (Research Question 1). Measures of 
teachers’ content- pedagogical knowledge were drawn from multiple choice and performance 
assessments, both of which were administered before the start of the project and at the 
conclusion of the impact study year. All measures were focused on the magnetism and 
electricity unit, as this was the only unit implemented by all teachers and the focus served to 
conserve limited resources, including teacher time. Described more thoroughly in Herman, 
Osmundson, & Dai, 2010, each of these measures is summarized below. 
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The multiple ehoice test coneentrated on three primary conceptual areas: magnetism, 
electricity, and electromagnetism; these were the three major concepts addressed in the study 
curriculum. Although composed of released National Assessment of Educational Progress 
(NAEP) and state assessment items — test reliability (coefficient alpha) was moderate, at .73 
for the total score. 

The performance-oriented assessment addressed teachers’ 
interpret student responses, a proxy for teachers’ pedagogical 
structure of these items was as follows: 

1. Teachers answered an open-ended content question related 
concepts; 

2. Student responses to the same question were provided, and teachers were asked to 
analyze and interpret the student responses; and 

3. Based on their analysis, teachers were asked to indicate the nature of specific students’ 
understandings and what they would do next to support student progress. 

Figure 1 shows a sample item that follows the teacher content survey sequence 
described above. 

1.22 Anne is investigating objects and magnets. She made this observation in her science Journal. 

"I was scjr’prised/ A nail was stock to the magnet 
When I accidentally touched the nail to a paper' 
clip, the paper clip stock to the nail, I wonder 
why that happened?" 

a. Explain to Anne why the paper clip stuck to the nail. Use diagrams or pictures if necessary. 




capacity to analyze and 
content knowledge. The 

to one of the three major 



Anne and her friend were asked by her teacher why they thought the paper clip stuck to the nail. 
Here are their responses to the question: 

Anne's response: The paper clip turned into a magnet too. 

Anne's friend's response: The nail gets stock on the magnet and the nail turns into a magnet 

so the paper clip can stick on the nail. 

b. What inferences can you draw about the students' understanding of magnetism and 
electricity? What do these students know? What do these students not know/need to learn? 

c. If these students were in your class, what would you do next in your instruction to help the 
students learning progress? 



Figure 1. Teacher content survey: Magnetism and electricity module. 
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Teacher responses to these performance items (Figure 1- parts b & c) were scored 
based on a 0-3 scale, derived from the defining features of expert ratings of a sample of 
teacher responses. A score of 0 was used for a non-response or irrelevant response, while 3 
reflected a complete and accurate description of student understandings and misconceptions 
or of next steps for instruction. Three raters participated in the scoring, all experienced 
science educators who were specially trained on the scoring rubric and familiar with the 
curriculum module. Pre- and post-test responses were scored together, with scorers blind to 
testing occasion. Based on a 25% sample of the responses that were double scored, reliability 
of scoring ranged between 76% agreement to 96% agreement (see Tables 2 & 3). 

Table 2 



Cohort 1 : Pre-survey Inter -rater Reliability, Open-ended 
Responses 



Comparison rater 


Rater 1 


Rater 2 


2 


0.96 






<.0001 




3 


0.90 


0.86 




<.0001 


<.0001 


Note. Pearson Correlation Coefficients, N = 
Prob > r under HO: Rho=0. 


63. 


Table 3 






Cohort 1 : Post-survey, Inter-rater Reliability, Open-ended 
Responses 


Comparison rater 


Rater 1 


Rater 2 


2 


0.86 






<.0001 




3 


0.91 


0.76 




<.0001 


<.0001 



Note. Pearson Correlation Coefficients, N~ 126. 
Prob > |r| under HO: Rho=0. 



Table 4 displays score reliabilities for the pre- and post-performance teacher 
assessment. Results show reasonable reliability for the analysis and interpretation and next 
step subscales, particularly given the small number of items constituting each. Scores for the 
content knowledge questions were less reliable than the other two areas, which in part may 
be due to the small number of items and potential ceiling effects (a total of seven items). 
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Table 4 



Cohort 1 Score Reliabilities for Performance Items on Content Survey 



Items 


Pre 


Post 


Content knowledge 


0.51 


0.48 


Analysis and interpretation 


0.73 


0.81 


Next instructional steps 


0.79 


0.84 



Teachers’ assessment practices (Research Question 2). Data on teachers’ assessment 
practices were derived from weekly online logs that educators completed as part of the study. 
The logs were originally designed as a fidelity of implementation measure for the larger 
study; they asked teachers to report on: (a) how much time they spent teaching the 
curriculum; (b) what instructional strategies they were using; (3) their use of available 
assessment tools and strategies; and (4) their evaluation of their students’ level of 
understanding relative to specific learning goals. While log completion rates varied greatly 
from teacher to teacher — ranging from some teachers completing as few as 2 logs and others 
completing more than 20 — data were available for almost all teachers in the study. 

As Table 5 reveals, factor analysis of the log data showed a clear intensity of 
assessment factor. Moreover, scores on this factor were significantly and positively related to 
classroom observation measures of assessment quality for the Magnetism and Electricity 
module. Correlation coefficients revealed a moderately strong relationship — .75 (for more 
detail, see Osmundson et ah, 2010). Thus, responses to items on this factor were used to 
construct a measure of teachers’ assessment practices for the current study (alpha=.95). 

Table 5 



Teacher Logs“ : Principle Component Analysis'’ 



Assessment component 


Aggregated items 
(Q_ave=Q_ave/Q 1 BSUM) 


Factor 1 
— asessmenf’ 


Used At a Glance 


Q2A_ave 


0.68 


Planned & used assessment 


Q3A_ave 


0.69 


Analyzed student work in notebook 


Q3b_ave 


0.79 


Analyzed student work on response sheets 


Q3c_ave 


0.72 


Analyzed observations of students 


Q3d_ave 


0.74 


Recorded and used assessment information on informal 
data chart 


Q3F_ave 


0.65 
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Assessment component 


Aggregated items 
(Q_ave=Q_ave/Q 1 BSUM) 


Factor 1 
— asessmenf’ 


Provided feedback to individual student based on analysis 
of student work 


Q3g_ave 


0.67 


Provided feedback to the entire class based on analysis of 
student work 


Q3I_ave 


0.64 


Retaught content 


Q3K_ave 


0.61 


Sum (#time curriculum taught/week) 


QIBSUM 


0.76 


Average (minute s/day analyzing student work) 


QIDAVE 


0.68 



“Treatment teachers only. 

'’Rotated Factor Pattern includes data from all three modules. 



Raw scores for items loading on the assessment faetors were eonverted into z seores 
and z seores were used to eompute a total -assessment faetor” seore for subsequent analysis 
in the path models. 

Student outcome measures (Research Question 3), The full study ineludes pre/post 
measures of student learning for eaeh of two modules eompleted for the study. The first is a 
speeially-developed end-of- year (EOY) assessment that addresses eore topies within the 
modules. The seeond is an end-of-year state assessment in English-language arts, 
mathematies, and scienee. However, seoring of the pre/post assessments is not yet complete. 

State assessment seores for English language arts (EEA) and math were available for 
both the year prior and the end of the study year, but the state seienee assessment was only 
implemented in grade 4 and thus available only for students for the end of the year in which 
they partieipated in the study. Prior year EEA and math seores were used as a eovariate, as 
were available data on student demographies. 

The EOY assessment was speeially developed by WestEd to address the eontent of the 
three modules that were part of the study: magnetism and eleetrieity, water, and strueture of 
life. (Reeall that eaeh teaeher implemented two modules, magnetism and eleetrieity plus one 
of the other two). Administered at the end of the study year, the assessment was eomprised of 
30 multiple-choiee questions, 10 on each of the three content areas. Based on the pilot 
version of the test, reliability was estimated at .76 (KR-20) and standard error of 
measurement based on KR-20 estimated at 2.57. 

Analysis 

Descriptive statisties and path analyses were used to examine the study’s primary 
researeh questions. Beeause the underlying study involved an assessment intervention. 
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observed differenees in effects on treatment and control teachers also were of interest. That 
is, because the intervention focused on the availability and use of curriculum-embedded 
assessment, any treatment effects on teachers might also suggest the impact of assessment 
use. 



Results 

Research Question 1: Teacher Knowledge 

Multiple-choice pre/post content survey results. The results presented in Tables 6 
and 7 display Cohort 1 control and treatment teachers’ performance on the multiple-choice 
post-test, and compares scores before and after participating in the study (i.e., after teaching 
the ASK/FOSS Magnetism and Electricity unit for two consecutive years). Results show that 
control and treatment teachers started the study with moderate scores and made gains in all 
areas of the content survey after two years of study participation. For the control group, the 
difference in pre/post scores is statistically significant at the .05 level for three of the four 
scales (magnetism and two electricity concepts). For the treatment group, the difference in 
the pre/post scores is statistically significant for all four scales, at the .05 level. For both 
groups, scores started the lowest and increased most on items relating to electromagnetism. 



Table 6 

Cohort 1 Pre/Post Magnetism and Electricity Content Survey Scores: Control Teachers 



Pre/post multiple choice scores 
control teachers 


Investigation 


N 


Pre 


Post 


Pre/post 


df 


t value 


Pr >|t| 


Magnetism 


11 


0.69 


0.83 


0.14 


10 


1.85 


0.093 


Electricity 1 


11 


0.68 


0.88 


0.20 


10 


4.9 


0.001 


Electricity 2 


11 


0.65 


0.87 


0.22 


10 


3.36 


0.007 


Electromagnetism 


11 


0.50 


0.82 


0.32 


10 


1.64 


0.1319 
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Table 7 

Cohort 1 Pre/Post Magnetism and Electricity Content Survey Scores: Treatment Teachers 



Pre/Post multiple choice scores 
treatment teachers 



Investigation 


N 


Pre 


Post 


Pre/post 


df 


t value 


Pr >|t| 


Magnetism 


13 


0.75 


0.96 


0.21 


12 


5.45 


0.000 


Electricity 1 


13 


0.68 


0.90 


0.22 


12 


5.42 


0.000 


Electricity 2 


13 


0.64 


0.90 


0.26 


12 


4.98 


0.000 


Electromagnetism 


13 


0.35 


0.92 


0.58 


12 


6.04 


<.0001 



Note. Unique teacher IDs were used to match teachers’ pre/post scores. Not all teachers completed pre/post 
content surveys, due to scheduling conflicts, hence the lower number of teacher scores reported than study 
participants. 

In general, we did not find statistieally signifieant differenees in teaehers’ post-test 
knowledge on the multiple-choiee test as a funetion of treatment eondition (eontrol vs. 
treatment). The results for the first subseale (magnetism) were an exeeption; in faet, when 
eontrolling for pre-test performance, treatment teachers outperformed control teachers on the 
multiple choice post-test items on magnetism. 

Content-pedagogical performance assessment results. Pre- and post-teacher scores 
for the content-pedagogical performance assessment are shown in Table 8. For both groups, 
teachers’ initial scores in content were modest, similar to the multiple-choice results, but 
scores on performance analysis and interpretation of student work and knowledge of 
instructional next steps were quite low, achieving on average only 29% to 37% of total 
possible points. While treatment teachers’ scores appeared slightly higher at the beginning of 
the study, the differences are not statistically significant. 

Similar to the multiple-choice trends, both groups also improved their scores on the 
content-pedagogical performance assessment after teaching the Magnetism and Electricity 
Module for two consecutive years. For both groups, the largest gain was in the area of 
instructional next steps. 
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Table 8 

Cohort 1 : Pre/Post Content-Pedagogical Performance Scores 







Control A= 13 






Treatment N=16 




Items 


Pre 


% Correct 


Post 


% Correct 


Pre 


% Correct 


Post 


% Correct 


Content^ 


4.1 


58.57 


5.4 


77.14 


4.4 


62.86 


6.6 


94.20 


Analysis and 


6.5 


30.95 


10.8 


51.43 


7.8 


37.14 


14.7 


70.00 


interpretation'’ 


















Instructional 


6.0 


28.57 


11.4 


54.29 


7.6 


36.19 


14.7 


70.00 


next steps“ 



















“Scale =1 (correct), 0 (incorrect), 7 possible points; '’Scale range 0-3 (see scales above), 21 possible points; 
“Scale range 0-3 (see descriptions above), 21 possible points. 



Regression analyses using the pre-survey seore for each question type as a covariate 
showed a statistically significant treatment effect for teachers’ content-pedagogical 
knowledge. For all three areas - content, analysis and interpretation, and instructional next 
steps - treatment teachers outperformed control teachers. Table 9 displays results of the 
regression analyses. 

Table 9 



Cohort 1 Regression Analysis for Pre/Post Content Survey 



Variable 


c/f 


Parameter estimate 


SE 


t value 


Pr > |t| 


Post content 


Intercept 


1 


5.15 


0.58 


8.96 


<.0001 


Pre content 


1 


0.06 


0.12 


0.46 


0.65 


Treatment 


1 


1.20 


0.39 


3.06 


0.01 


Post analysis & interpretation 


Intercept 


1 


8.77 


1.44 


6.11 


<.0001 


Pre analysis & interpretation 


1 


0.31 


0.18 


1.73 


0.10 


Treatment 


1 


3.50 


1.20 


2.92 


0.01 


Post next step 


Intercept 


1 


9.16 


1.46 


6.27 


<.0001 


Pre next step 


1 


0.37 


0.19 


1.98 


0.06 


Treatment 


1 


2.73 


1.31 


2.08 


0.05 



12 




Research Question 2: Assessment Implementation 

Log results. Table 10 summarizes the deseriptive results using the teaeher as the unit 
of analysis and mean seores for eaeh item over the eourse of the unit. While average 
responses were generally similar between treatment and eontrol teaehers, it is noteworthy 
that treatment teachers spent significantly more time looking at student work and were more 
likely to report spending more than 10 minutes a day in doing so than control teachers. 

On average, the results reported in Table 11 indicate that teachers used the study 
modules three times a week and, on each of these days, reported spending 5-10 minutes 
analyzing student’ work on the modules. Teachers reported that they most frequently 
analyzed their observations of students, roughly half the days they taught the modules. Other 
assessment activities - provision of individual feedback to students, analysis of student 
written work in notebooks or response sheets - occurred with relatively less frequency. 
While there is considerable variability in these scores, it is consistent with the variability in 
the number of times a week teachers reported using the modules - that is, if teachers used the 
modules for science four times a week, there was the possibility of engaging in each 
assessment activity four times, while if the modules were used three times a week, the 
frequency of potential assessment use would be reduced accordingly. 
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Table 10 

2009-2010: Cohort 1 Teacher Log Data Descriptive Results (All Modules) 



Teacher log questions 


Cohort 1 : Control 
A=14 teachers 


Cohort 1: Treatment 
N=\6 teachers 


Number of times/week used modules 


2.9 


3.1 




(0.9) 


(0.8) 


Assessment time 


Average minutes/day looking at student work 


5.9 


10.8 




(5.8) 


(6.9) 


Percentage of logs where teachers reported spending 


20% 


43% 


more than 10 minutes/day looking at student work 


(0.3) 


(0.4) 


Use of assessments^ 


Provided feedback to individual students based on 


1.2 


1.2 


analysis of student work 


(0.8) 


(0.7) 


Analyzed observations of students 


1.6 


1.6 




(1.1) 


(0.8) 


Checked on student understandings at the end of an 


1.4 


0.8 


investigation 


(0.9) 


(0.5) 


Engaged students in self-assessment of science 


1.0 


0.7 


learning 


(1.1) 


(0.5) 


Analyzed student work in science notebooks 


1.1 


1.4 




(1.1) 


(0.8) 


Analyzed student work on student response sheets 


1.3 


1.1 




(0.8) 


(0.8) 



“Scale = Number of times/week teacher reported engaging in activities. 



Table 1 1 shows factor scores on assessment implementation for control and treatment 
teachers. Results suggest that treatment teachers’ logs revealed significantly greater 
implementation of assessment than did control teachers, as judged by time spent analyzing 
student work and use of various strategies. 

Table 1 1 



Assessment Implementation Factor Scores for Cohort ITreatment and Control Teachers 



Group 


Assessment Implementation Scores 


N 


Control 


-0.82 


14 


Treatment 


-0.07 


16 


Difference treatment/control 


0.75* 





* Statistically significant at the alpha <0.05 level. 
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Research Question 3: Student Outcomes 



Entering abilities. Student state assessments in reading results at the end of the year, 
which were taken prior to the study, provide a gauge of students’ entering ability. Table 12 
summarizes students’ mean scale scores; these scores indicate that, on average, students 
participating in the study were performing at the level of ^neeting state standards” in 
reading. 



Table 12 



Third Grade State Assessment Results in Reading for 
Cohort 1 Treatment and Control Students 



Group 


N 


Mean 


SD 


Treatment 


400 


470.10 


51.32 


Control 


314 


465.50 


51.95 



State assessment and end-of-year science results. Table 13 summarizes classroom 
results for the end-of-year science test (developed especially for the study) as well as the 
state assessment in science. Both tests were administered to students at the completion of the 
study year. The data show that, on average, students achieved about 60% correct on the end- 
of-year measure. 



Table 13 

End-of-Year Science Assessment Scores for Cohort 1 Treatment and Control Classrooms 







Treatment students 






Control students 


Test 


N 


Mean 


SD 


N 


Mean 


SD 


End-of-year science assessment 


410 


19.18 


4.45 


323 


18.28 


5.04 


State assessment: Science 


435 


533.00 


54.47 


344 


522.40 


51.82 



Path Analysis Results 

Path analysis using the combined data from the treatment and control groups was 
utilized to test the relationships among and between teachers’ initial content-pedagogical 
knowledge, assessment practices, and student learning. The path model controlled for 
students’ entering ability and predicted that teachers’ assessment practices and student 
learning would be directly related, but that teachers’ content knowledge would have only an 
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indirect relationship to student learning through its influence on teachers’ assessment 
practices. That is, teachers with higher content-pedagogical knowledge were expected to 
engage in more use of assessment, and greater use of assessment was expected to be 
positively related to student learning, but no direct relationship was expected between 
teachers’ content-pedagogical knowledge and student learning. 

Because the teacher measures were highly related and produced similar models, a 
composite measure of teachers’ content-pedagogical knowledge was used, which combined 
results across the multiple choice and performance assessment measures at the start of the 
study (i.e., pre-test). Models were tested using both the end-of-year student assessment and 
state science assessment results as the indicator of student learning. Moreover, in order to 
examine the extent to which teachers’ use of assessment might positively influence their 
content-pedagogical knowledge, analyses also examined the relationship between changes in 
teachers’ content-pedagogical knowledge and assessment practices. All models were tested at 
the teacher level and used students’ prior year grade) state reading scores to control for 
any differences in entering student ability. 

Figure 2 displays the standardized path coefficients evident in the relation between the 
students’ entering ability, based on standardized test results in reading; a composite measure 
of teachers’ content-pedagogical knowledge at the start of the study; teachers’ assessment 
use, as measured by weekly logs; and student performance on the end-of-year science 
assessment. The path coefficients show how much a change of one standard deviation in a 
prior variable would produce in standard deviation units of the subsequent variable. Results 
show that, after controlling for students’ prior ability, teachers’ assessment use is 
significantly and positively related to students’ end-of-year performance. A change of one 
standard deviation in teachers’ assessment use scores is associated with a change of .26 
standard deviation in students’ performance. Teachers’ content-pedagogical knowledge has 
no direct relationship to student learning, but indirectly affects it through a marginally 
significant relationship to teachers’ assessment use. That is, stronger content-pedagogical 
knowledge is marginally and positively related to teachers’ assessment use, which is 
positively related to student learning. Bentler’s Comparative Fit Index show a good fit for the 
model displayed (CFI=.9865). Analyses are ongoing and use students’ end-of-year scores on 
the state science assessment. 
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Path Model Results 




DV: Dependentvariable 
IV: Interdependentvariable 

a#: Estimated standardized path coefficient among observed variables 

*: Significant @ p<. 05 

®: Significant@ p<.l 

Figure 2. Standardized path coefficient model. 

Study analyses found no relationship between ehanges in assessment use and ehanges 
in teacher practices. That is, teachers who spent more time in assessing and responding to 
students’ work did not gain more content-pedagogical knowledge over the course of the 
study than those who spent less time. 



Discussion 

This paper started with three research questions. We end it by summarizing our 
findings with regard to each question and then consider implications. 

Research Question 1: What is the quality of teachers’ content-pedagogical knowledge? 

The study used multiple-choice and performance assessments to measure teachers’ 
content and pedagogical knowledge. The multiple choice test was drawn from publically 
available items on magnetism, electricity, and electromagnetism intended for elementary 
students (which sets an admittedly low bar for teachers’ content knowledge). At the 
beginning of the study teachers scored from 35% to 75% correct. By the end of the study. 
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after having taught the study eurriculum twiee, performance ranged from 82% to 92% 
correct. 

While results of the performance assessment showed the same positive trends from pre- 
to post-study, the levels of performance were less promising, particularly for tasks on the 
analysis and interpretation of student work and knowledge of instructional next steps, where 
scores were quite low. Scores on the pre-test ranged from 29% to 37% of total possible 
points and from 45% to 54% on the post-test. For both treatment and control groups, the 
largest gain was in the area of instructional next steps. 

The fact that the performance assessment actually engaged teachers in formative 
assessment (i.e., analyzing student work and identifying implications for subsequent 
instruction) also bears directly on participating teachers’ assessment capacity. Consistent 
with prior literature (Herman et ah, 2010; Heritage et al. 2009), results suggest limited 
teacher capacity. 

Research Question 2: What is the relationship between teacher knowledge and 
assessment practice? 

Because the study treatment focused on systematically embedding formative and end- 
of-investigation assessments in a hands-on science curriculum and encouraging teachers to 
regularly analyze student work, observed differences in the relative gains in content- 
pedagogical knowledge for treatment relative to control teachers were suggestive of the 
effects of sound assessment use on teacher knowledge. That is, regression analyses 
controlling for teachers’ pre-study performance showed a statistically significant treatment 
effect for teachers’ content-pedagogical knowledge at the completion of the study, which 
might be related to the treatment’s stronger focus on assessment and the quality of those 
assessments. For all three areas - content, analysis and interpretation, and instructional next 
steps - treatment teachers outperformed control teachers. 

However, path analyses examining the relationship between teachers’ assessment use 
and changes in teachers’ content knowledge revealed no statistically significant relationships. 
Higher use of assessment, as measured by teachers’ responses to weekly logs, was not 
associated with stronger teachers’ content-pedagogical knowledge. 

Research Question 3: What is the relationship between teacher knowledge, assessment 
practice and student learning? 

Path analysis results supported Black and Wiliam’s conclusions (1998) and the paper’s 
hypothesis about the relationship between teachers’ use of assessment in instruction and 
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student learning. Controlling for students’ entering ability, as measured by standardized test 
scores in reading, the manner in which teachers utilized assessment was positively related to 
student learning outcomes. More use of assessment was associated with higher student 
performance at the end of the study. As expected, teachers’ initial content-pedagogical 
knowledge, as gauged by multiple measures administered at the start of the study, showed no 
direct influence on student learning, but was marginally related to teachers’ assessment use. 
Admittedly, however, the relationship was of little practical significance. The overall model 
fit was very high for these analyses. 

Conclusions 

Study findings reinforce the power of formative assessment, or at least one important 
element of it: Students whose teachers spend more time and who more frequently engage in 
analyzing and providing feedback on student work achieve higher learning than students 
whose teachers spend less time and who less frequently do so. Teachers’ attention to student 
learning as evidenced in classroom work — whether through observations of students in 
classroom discussions or analyses of student responses in science notebooks, other written 
responses, or end-of-investigation assessments — is associated with higher student 
performance. 

The strength of this relationship is striking in light of the weaknesses in teachers’ initial 
content-pedagogical knowledge, as documented in pre-test scores for this study. It seems 
obvious that sound formative assessment practice requires adequate content-pedagogical 
knowledge. In other words, it is hard to imagine how teachers with weak knowledge of 
subject matter content and of the nature of students’ progression through the content can 
appropriately analyze student work, or make appropriate decisions for next steps. Path 
analysis results from this study weakly support this supposition, as teachers’ content 
knowledge showed an indirect relationship with student learning through teachers’ use of 
assessment. 

That both treatment and comparison teachers showed substantial pre- to post- 
intervention gains may be at least partially due to a testing effect (the same assessment was 
given pre-intervention and two years later, at the end of the study). However, since the 
treatment group showed substantially higher content-pedagogical knowledge at the end of the 
study, it is possible that the use of sound assessment tools, as embedded in the intervention, 
may contribute to the development of stronger teacher knowledge. While the path model 
examining changes in teachers’ knowledge did not show this connection, the small sample 
size, particularly of the treatment group, may have been an obstacle. The addition of Cohort 2 
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data from our study will broaden the sample for addressing these issues and form clear next 
steps for us. 

We conclude with the same chicken or egg problem that we used in titling the paper. 
Teachers’ use of formative assessment benefits student learning, as the findings reported here 
substantially document. Yet, effective formative assessment places heavy demands on 
teachers’ content-pedagogical knowledge — knowledge that may be spotty based on the 
present’ study’s findings as well as other research. Can analyzing student responses to sound 
assessments help teachers strengthen their content-pedagogical knowledge? Is a minimal 
level of knowledge necessary for effective assessment use that benefits learning? What are 
optimal approaches for developing teachers’ capacity in these areas? The current study raises 
possibilities, but the chicken-egg issue remains unresolved. The omelet remains on the table, 
which continues to be a popular entree in current policy initiatives. The present study’s 
results underscores both the potential and challenge of bringing these initiatives to fruition. 
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TEACHER CONTENT SURVEY 




STUDY 



PURPOSE OF THIS INSTRUMENT 

This measure is designed to collect information about teacher understandings of magnetism and 
electricity and approaches teachers use to understand student thinking. Results from the survey 
will help us to better understand how FOSS works to help students learn science. 



INSTRUCTIONS 

I.You have been alloted 30-45 minutes to complete this measure. However, if you wish, 
you may use more time during your break in order to finish it. You may choose to not answer 
questions and/or stop your work at any point during the time period. 

The content survey includes questions with a wide range of difficulty, and we expect you to 
encounter items for which you may not know the answers. If you are not sure of an answer, 
please make your best guess — there is no penalty for guessing. 



2. Please fill in your name and ID numbers below and your ID on the next page. 



First name 



Last name 



Date 



Your ID Number: 



T 



IMPORTANT: 

To keep your data confidential, this cover sheet with your name will be removed upon 
receipt by the research staff, leaving only your ID number on the next page of the survey. 
This cover sheet will be stored in a locked cabinet, separate from the completed surveys. 



SECTION 1 



Your ID Number: 



T 



1.1 1 Julie placed a paper clip, piece of cardboard, and magnet together like you see in the picture. 




Why did the paper clip stay in place next to the cardboard instead of falling to the floor? 
Choose the best answer. 

O A. The paper clip is made of iron and so is the magnet. 

O B. The magnetic field goes around the cardboard and makes the paper clip stay there. 
O C. The magnet has a magnetic field that is not blocked by the cardboard. 

O D. The electric force field makes the paper clip attract to the magnet. 

1.12 Arthur was playing with magnets. He had one magnet on the table, and one in his hand. 
As he moved the magnet in his hand closer to the one on the table, the magnets suddenly 
snapped together. 

a. Explain why the magnets snapped together even though they were not touching. 
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SECTION 1 



TEACHER CONTENT SURVEY 



Here are two students' responses to question 1.12: 

Student 1 Response: Both magnets ar^e made of ir’on, and the magnets ar’e both facing 

south and south. 

Student 2 Response: The magnets snapped together’ because the electr’ic fields got close. 

b. What inferences can you draw about the students' understanding of magnetism and 

electricity? What do these students know? What do these students not know/need to learn? 



c. If these students were in your class, what would you do next in your instruction to help the 
students learning progress? 



1.21 A nail that was stuck to a permanent magnet picked up a small metal washer. The nail could pick 
up the metal piece because: 

OA. Nails have magnetic fields. 

O B. Magnetism was induced in the nail. 

O C. The nail and the washer are both made of iron. 

O D. The washer is still in the range of the magnetism. 
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SECTION 1 



TEACHER CONTENT SURVEY 



1 .22 Anne is investigating objects and magnets. She made this observation in her science journal. 




“I was surpr’isedl A nail was stuck to the magnet 
When I accidentally touched the nail to a paper’ 
clip, the paper clip stuck to the nail. I wonder 
why that happened?” 



a. Explain to Anne why the paper clip stuck to the nail. Use diagrams or pictures if necessary. 



Anne and her friend were asked by her teacher why they thought the paper clip stuck to the nail. 
Here are their responses to the question: 

Anne's response: The paper clip turned into a magnet too. 

Anne's friend's response: The nail gets stuck on the magnet and the nail turns into a magnet 

so the paper clip can stick on the nail. 

b. What inferences can you draw about the students' understanding of magnetism and 

electricity? What do these students know? What do these students not know/need to learn? 



c. If these students were in your class, what would you do next in your instruction to help the 
students learning progress? 
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SECTION 1 



TEACHER CONTENT SURVEY 



1.31 a. Complete the following table. Put an'X'in the second column of the table if the object 

sticks to a magnet. Put an'X'in the third column of the table if the object conducts electricity. 



Object 


Sticks to a magnet 


Conducts electricity 


Iron nail 






Plastic straw 






Steel wire screen 






Wooden craft stick 






Brass ring 






Rubber band 






Copper penny 






Piece of aluminum foil 







b. Why did you choose the objects that you did in the"Sticks to a magnet" column? 
Use diagrams or pictures to show your thinking. 



c. Why did you choose the objects that you did in the"Conducts electricity" column? 
Use diagrams or pictures to show your thinking. 
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SECTION 1 



TEACHER CONTENT SURVEY 



1.32 Here is how one student completed the table. 



Object 


Sticks to a magnet 


Conducts electricity 


Iron nail 


X 


X 


Plastic straw 






Steel wire screen 


X 


X 


Wooden craft stick 






Brass ring 


X 


X 


Rubber band 






Copper penny 


X 


X 


Piece of aluminum foil 


X 


X 



Here are one student's responses to questions 1 .3 1 b and 1 .3 1 c (see page 5): 

Student 1 Response: 

1.31 b. These things stick to the magnet because they are all metal. 

1.31 c. These things are all made of metal and metal conducts electricity. 

a. What inferences can you draw about the students' understanding of magnetism and 

electricity? What do these students know? What do these students not know/need to learn? 



b. If these students were in your class, what would you do next in your instruction to help the 
students learning progress? 
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SECTION 1 



TEACHER CONTENT SURVEY 



1 .41 The picture below shows Maria pushing Magnet 1 toward Magnet 2 on a smooth table. 
Both magnets are lying on a smooth table. 




a. What will happen as Magnet 1 moves towards Magnet 2? 



b. Why will this happen? 
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SECTION 1 



TEACHER CONTENT SURVEY 

1 .42 Three bar magnets are held together as shown in the picture below. 

Magnet 1 Magnet 2 Magnets 

I II II 1 




a. What will the magnets do when they are released? Circle the correct answer. 



/N 


7 


/S / N 


/s 


/ N/ 













^ 




b. Why does that happen? 
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SECTION 1 



TEACHER CONTENT SURVEY 



1.46 Lisa found a magnet with no labels on the poles. She found another magnet with correctly 
labeled poles and put the magnets together. They attracted. 



s 


7 

• 






N 





a. The pole labeled with the "?" is most likely which pole? 

O A. south pole 
O B. north pole 

O C. not enough information provided 

b. Why? Please explain your answer. 
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SECTION 1 



TEACHER CONTENT SURVEY 



WRAPPING IT UP 

1 . What is/are the key concept/s addressed by the assessments in Section 1 ? 



2 . Why is it important for students to learn these magnetism and electricity concepts? 
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SECTION 2 



2.1 1 Look at the picture below. What kind of circuit is this? 




How do you know? 



OA. network circuit 

OB. series circuit 

OC. parallel circuit 

OD. short circuit 



2.1 2 Look at the picture below. What kind of circuit is this? 

OA. simple circuit 

OB. series circuit 

OC. parallel circuit 

OD. short circuit 



How do you know? 



2.1 3 Look at the picture below. What kind of circuit is this? 

OA. simple circuit 

OB. series circuit 

OC. parallel circuit 

OD. short circuit 



How do you know? 
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SECTION 2 



TEACHER CONTENT SURVEY 



2.1 4 Look at the picture below. What kind of circuit is this? 




OA. simple circuit 

OB. network circuit 

OC. series circuit 
O D. parallel circuit 



2.1 5 a. Draw in lines representing wires to make a parallel circuit. 




Explain your drawing: what features make this a parallel circuit? 
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SECTION 2 



TEACHER CONTENT SURVEY 



2.21 Look at the picture below. 




a. Will the bulb light? OYes ONo 

b. Is the circuit complete? OYes ONo 



2.22 Look at the picture below. a. Will the bulbs light? OYes ONo 




2.23 Look at the picture below. 




a. Will the bulb light? OYes ONo 

b. Is the circuit complete? OYes ONo 



C. Explain why you think the circuit is or is not complete. 
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SECTION 2 



TEACHER CONTENT SURVEY 



2.24 Look at the picture below. The round object in the middle of the picture is an empty bulb holder. 



a. Will the bulb light? OYesONo 

b. Explain why you thinkthe bulb will or will not light. 



This is how a Student 1 responded to question 2.24. 

a. Will the bulb light? O Yes ® No 

b. Bulb won't light because it's not connected to the batter’y. 

This is how a Student 2 responded to question 2.24. 

a. Will the bulb light? O Yes® No 

b. Bulb won't light because it's a shor’t cir’cuit 

c. What inferences can you draw about the students' understanding of magnetism and 
electricity? What do these students know? What do these students not know/need to learn? 



d. If these students were in your class, what would you do next in your instruction to help the 
students learning progress? 
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SECTION 2 



TEACHER CONTENT SURVEY 



2.31 



Draw arrows on the picture to show which direction electricity will flow through the circuit to run 
the motor. 

Explain your answer. 




2.32 Denise wants to build a circuit that will light up a bulb and run a motor at the same time. 

She drew the diagram of the circuit she planned to build. She used a special switch in the circuit. 
The switch is shown in the gray box. 




a. Look at the diagram Denise drew. Explain to her why you think her circuit would or would 
not work the way she wants it to work. 
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SECTION 2 



TEACHER CONTENT SURVEY 



2.33 Below are two student's responses to question 2.32. 

Student 1 response: I think it would wor’k because all the par’ts of connected. But it might 

not wor’k because the battery might not have enough juice to carry 
all on one circuit 

Student 2 response: It probably won't because the energy can't go two different ways. 

a. What inferences can you draw about the students' understanding of magnetism and 
electricity? What do these students know? What do these students not know/need to 
learn? 



b. If these students were in your class, what would you do next in your instruction to help 
the students learning progress? 



2.41 Electricity can be changed into other forms of energy. Complete the sentences below: 

a. The bulb in a lamp changes electric energy into 

b. A motor changes electric energy into 
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SECTION 2 



TEACHER CONTENT SURVEY 



2.42 Which of the following items converts electric energy into motion? 

O A. light switch 

O B. electric stove 
O C. light bulb 
O D. electric fan 

2.43 When an electric stove is turned on, most of the incoming electrical energy changes into: 
O A. heat energy 

O B. light energy 
O C. mechanical energy 
O D. sound energy 

2.44 Which of the following items converts electric energy into light? 

O A. light switch 

O B. doorbell 
O C. light bulb 
O D. electric fan 

2.45 When an electric fan is running, most of the incoming electric energy is converted into: 

O A. heat energy 

O B. light energy 
O C. motion energy 
O D. sound energy 

2.46 Household appliances convert electricity into one or more different forms of energy. 

An electric fan can best be described as converting electricity into: 

O A. heat energy only 

OB. heat energy, and sound energy 

O C. heat energy, sound energy and motion energy 

O D. heat energy, sound energy, motion energy and chemical energy 
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SECTION 2 



TEACHER CONTENT SURVEY 



WRAPPING IT UP 

1 . What is/are the key concept/s addressed by the questions in Section 2? 



2 . Why do students need to know these concepts about magnetism and electricity? 
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SECTION 3 



3.1 1 Annie had three rivets. One was copper, one was iron and one was steel. Which rivet or rivets 
could she use to make an electromagnet? Why? 



3.1 2 Here are two students' response to question 3.1 1 

Student 1: Annie should use the iron and steel rivets because they conduct electricity 
and they stick to magnets. 

Student 2: Annie could use the iron, copper or steel rivets because they are all metal. 

a. What inferences can you draw about the students' understanding of magnetism and 

electricity? What do these students know? What do these students not know/need to learn? 



b. If these students were in your class, what would you do next in your instruction to help 
the students learning progress? 
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SECTIONS 



TEACHER CONTENT SURVEY 



3.21 Imagine you have the following materials: a large iron nail, several permanent magnets, lots of 
insulated wire, a D-cell and a switch. 

a. Describe one way to make the nail a temporary magnet. 



b. Describe another way to make a temporary magnet. 



3.31 Samuel Morse, the inventor of the telegraph, had a problem. His telegraph's signal was too weak. 
He needed a stronger electromagnet. What are two ways he might have used to increase the 
strength of the electromagnet for his telegraph? 



20 



SECTIONS 



TEACHER CONTENT SURVEY 



3.41 Wendy is making an electromagnet. First, she wrapped a long, insulated wire around an iron nail. 
What should Wendy do to complete the electromagnet? 



Here are two student responses to question 3.41 : 

Student 1: Attach the wire to the D-cell and switch, nub the magnet on the nail a few 
times and then try it 

Student 2: Wendy should connect the iron nail to the D-cell to make a complete circuit 

a. What inferences can you draw about the students' understanding of magnetism and 

electricity? What do these students know? What do these students not know/need to learn? 



b. If these students were in your class, what would you do next in your instruction to help the 
students learning progress? 



3.42 Which of the following materials is NOT necessary to build an electromagnet? 
O A. a magnet 
O B. a steel rivet 
O C. a D-cell battery 
O D. wire 
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SECTIONS 



TEACHER CONTENT SURVEY 



WRAPPING IT UP 

1 . What is/are the key concept/s addressed by the assessments in Section 3? 



2 . Why do students need to know these concepts about magnetism and electricity? 
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